Fully or highly automated driving systems are designed to operate a vehicle on the road without driver interaction or other external control, for example, self-driving vehicles or autonomous vehicles. As part of safe operation in the autonomous mode, some autonomous vehicles are configured to detect nearby emergency vehicles within one or more images captured by a perception system. Other autonomous vehicles can infer that an emergency vehicle is nearby based on the behavior of surrounding vehicles responding to the emergency vehicle. However, the aforementioned examples of detection and response to an emergency vehicle are not sufficient in the case of an obstructed view of the emergency vehicle from the vantage point of the perception system of the autonomous vehicle or the lack of other vehicles surrounding the emergency vehicle.